{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "(vis-narrative)=\n",
    "# Narrative Data Visualisation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Introduction\n",
    "\n",
    "In this chapter, we'll examine the tricks and techniques of *narrative data visualisation*. This type of visualisation gets a lot more press than the others discussed in {ref}`vis-intro` ; literally, because journalists use it in their work. There are many books written on it too. \n",
    "\n",
    "For narrative visualisation, it's particularly helpful to bear this quote in mind:\n",
    "\n",
    "> The purpose of visualisation is insight, not pictures\n",
    "\n",
    "—Ben Shneiderman, populariser of the highlighted text link\n",
    "\n",
    "Narrative data visualisation requires the most thought in the step where you go from the first view to the end product. It's a visualisation that doesn't just show a picture, but gives an insight.\n",
    "\n",
    "Let's import the packages we'll need for the rest of the chapter."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import pandas as pd\n",
    "import matplotlib.pyplot as plt\n",
    "import matplotlib.ticker as ticker\n",
    "import matplotlib.dates as mdates\n",
    "import seaborn as sns\n",
    "\n",
    "# Set seed for reproducibility\n",
    "np.random.seed(10)\n",
    "# Set max rows displayed for readability\n",
    "pd.set_option(\"display.max_rows\", 6)\n",
    "# Plot settings\n",
    "plt.style.use(\n",
    "    \"https://github.com/aeturrell/coding-for-economists/raw/main/plot_style.txt\"\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Narrative Data Visualisation\n",
    "\n",
    "As discussed, the name of the game here is to communicate a particular narrative. Let's see what tricks we can use to help do this. The three stages of this are: deciding the story, deciding the chart, and creating a chart that helps deliver that narrative.\n",
    "\n",
    "### Deciding the story\n",
    "\n",
    "We are only human, and a digestible narrative goes a lot further than lots of data with no thread running through. If you are doing narrative visualisation, you must first be clear about the story you want to tell and why it's important. Let's say you are making a visualisation on topic 'Y', then some reasons why 'Y' might be important that you may want to think about when creating your narrative are:\n",
    "\n",
    "- Y matters: when Y rises or falls, people are hurt or helped.\n",
    "- Y is puzzling: it defies easy explanation.\n",
    "- Y is controversial or political: some argue one thing while others say another.\n",
    "- Y is big (like the service sector) or common (like traffic jams).\n",
    "- Y helps someone do something they could not do before.\n",
    "\n",
    "If you can identify why a story is important, you're half way to designing a narrative visualisation that brings the story into focus. Later on, we'll recreate a chart from the *Financial Times*. In that case, 'Y' is a high for air pollution in Beijing that hurts people (it's also political because of efforts to tackle the problem). So, in this case, the creator of the narrative needs to convey that the pollution has hit a high (presumably relative to other points in time) and that this high is far above safe levels.\n",
    "\n",
    "### What plot should I use?\n",
    "\n",
    "Once you know what story you want to tell, you need the right kind of chart for the job. Resources like the *Financial Times*' [visual vocabulary](http://ft-interactive.github.io/visual-vocabulary/) are extremely useful here. You need to ask yourself what element you're trying to highlight: a point in time, the size relative to other units cross-sectionally, the distribution either in numbers or spatially, the difference between groups, how something has changed, etc.? There are charts that can help with all of these and it's well worth looking at the link to get a sense. You can find code for many of the plots that are featured in the {ref}`vis-common-plots` chapter.\n",
    "\n",
    "### Drawing Attention to Enhance Narrative Visualisation\n",
    "\n",
    "According to data visualisation master Jon Schwabish's book *Better Data Visusalizations* {cite}`schwabish2021better`, there are 15 ways to draw an audience's attention in a chart:\n",
    "\n",
    "1. Shape\n",
    "2. Enclosure\n",
    "3. Line width\n",
    "4. Saturation\n",
    "5. Colour\n",
    "6. Size\n",
    "7. Markings\n",
    "8. Orientation\n",
    "9. Position\n",
    "10. Sharpness\n",
    "11. Length\n",
    "12. 3D\n",
    "13. Curvature\n",
    "14. Density\n",
    "15. Closure\n",
    "\n",
    "Sometimes people add a 16th entry to this list, Connection.\n",
    "\n",
    "![Preattentive Visual Processing, by @jschwabish.](https://github.com/aeturrell/coding-for-economists/raw/main/img/preattentive.png) The different ways to trigger preattentive visual processing, by @jschwabish.\n",
    "\n",
    "But be warned: not all of these are equivalent! It's much easier to perceive differences in length than it is differences in, say, volume. So if you want your audience to be able to make comparisons or quantitative assessments, you need to pick what techniques you use from this list carefully. Roughly in order of how easy they are to perceive quantitatively, the features are: one common axis, two axes, length, slope, angle, parts of whole, area, volume, saturation, and hue."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### The Final Mile in Narrative Visualisations\n",
    "\n",
    "Good visualisation helps the viewer to grasp the narrative—rather than leaving them puzzling as to what the key message is. To that end, various adornments may be added to a plot to bring out the narrative. These adornments typically take the form of those we have seen already that draw the eye, but for a successful narrative visualisation they must come together to tell a story.\n",
    "\n",
    "We'll also make use of some other tricks:\n",
    "\n",
    "- Text annotations, which can be a useful addition to a chart because they further enhance the narrative.\n",
    "\n",
    "- Declutter the graph, removing lines that aren't helping frame the story\n",
    "\n",
    "- Use the title to tell the story, and put the y-axis label horizontally below\n",
    "\n",
    "- Use faded text for text that isn't contributing directly to the narrative\n",
    "\n",
    "- If there are multiple lines, label them directly rather than via a legend\n",
    "\n",
    "Let's see an example that brings together quite a few of these elements, recreating a chart from the *Financial Times*: a newspaper that is well-known for its impressive visualisations. The chart tells the story of extremely high levels of air pollution in Beijing at the start of 2021. (Note that the data here disagree with the original Financial Times source, which were unavailable; do not take the numbers too seriously.)\n",
    "\n",
    "Let's first grab the data:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df = pd.read_csv(\n",
    "    \"https://github.com/aeturrell/coding-for-economists/raw/main/data/beijing_pm.csv\",\n",
    ")\n",
    "df[\"date\"] = pd.to_datetime(df[\"date\"])\n",
    "df = df.set_index(\"date\")\n",
    "# Restrict to time scale of interest\n",
    "df = df[(df.index >= \"2020-02-28\") & (df.index <= \"2021-03-01\")]\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now let's get on to the figure. The code that generates individual parts of the chart is annotated to explain what it's doing."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# set a level for fading some elements of the plot\n",
    "fade_alpha = 0.6\n",
    "\n",
    "fig, ax = plt.subplots()\n",
    "# Plot a line\n",
    "ax.plot(df.index, df[\"pm25\"], lw=1.5, color=\"#12549a\")\n",
    "# Title that gives the narrative\n",
    "plt.suptitle(\n",
    "    \"Severe dust storm causes sharp rise in Beijing's air pollution\",\n",
    "    size=12,\n",
    "    ha=\"left\",\n",
    "    x=0.12,\n",
    ")\n",
    "# Horizontal y-axis title, faded\n",
    "ax.set_title(\n",
    "    \"PM2.5 rolling 7-day average (micrograms per cubic metre)\",\n",
    "    loc=\"left\",\n",
    "    size=10,\n",
    "    alpha=fade_alpha,\n",
    ")\n",
    "# Time is obvious, so no x-label needed: instead annotate sources\n",
    "ax.set_xlabel(\n",
    "    \"* Based on annual mean exposure of 10 micrograms per cubic metre \\n Source: AQICN\",\n",
    "    loc=\"left\",\n",
    "    size=6,\n",
    "    alpha=fade_alpha,\n",
    ")\n",
    "# remove chart clutter\n",
    "for key, spine in ax.spines.items():\n",
    "    spine.set_visible(False)\n",
    "ax.tick_params(axis=\"y\", which=\"both\", length=0)\n",
    "ax.tick_params(axis=\"x\", which=\"both\", color=[1, 0, 0, fade_alpha])\n",
    "# set aesthetically pleasing limits\n",
    "ax.set_ylim(0, 200)\n",
    "ax.set_xlim(None, df.index.max())\n",
    "# for time series, tick marks on the right help give a sense of right-ward motion\n",
    "ax.yaxis.tick_right()\n",
    "# create grid only in y-direction, so viewer can judge level-but with few ticks\n",
    "ax.yaxis.set_major_locator(ticker.MaxNLocator(4))\n",
    "ax.grid(which=\"major\", axis=\"y\", lw=0.2)\n",
    "# add minor ticks for months, x-axis\n",
    "ax.xaxis.set_minor_locator(mdates.MonthLocator())\n",
    "# major ticks for quarters, x-axis\n",
    "ax.xaxis.set_major_locator(mdates.MonthLocator(bymonthday=1, interval=3))\n",
    "# label x-axis in Jan 01 format at major ticks\n",
    "ax.xaxis.set_major_formatter(mdates.DateFormatter(\"%b %y\"))\n",
    "# add an annotation to the \"news\" here, the latest peak. Use an arrow.\n",
    "ax.annotate(\n",
    "    \"Worst storm in a decade hits Beijing\",\n",
    "    xy=(df.idxmax(), df.max()),\n",
    "    xycoords=\"data\",\n",
    "    xytext=(-30, -40),\n",
    "    textcoords=\"offset points\",\n",
    "    ha=\"right\",\n",
    "    size=9,\n",
    "    arrowprops=dict(arrowstyle=\"->\", connectionstyle=\"angle3\"),\n",
    ")\n",
    "# Add a hatch, with a label, that represents the WHO safe level: this helps\n",
    "# put the current rise in context\n",
    "ax.fill_between(\n",
    "    x=df.index,\n",
    "    y1=0,\n",
    "    y2=10,\n",
    "    hatch=\"///\",\n",
    "    facecolor=\"None\",\n",
    "    linewidth=0.1,\n",
    "    alpha=0.2,\n",
    ")\n",
    "ax.annotate(\n",
    "    \"WHO safe level*\",\n",
    "    fontweight=\"heavy\",\n",
    "    xy=(0.6, 0.01),\n",
    "    xycoords=\"axes fraction\",\n",
    "    xytext=(0, 0),\n",
    "    textcoords=\"offset points\",\n",
    "    ha=\"right\",\n",
    "    size=9,\n",
    ")\n",
    "# Use the FT background colours\n",
    "fig.set_facecolor(\"#fff1e4\")\n",
    "ax.set_facecolor(\"#fff1e4\")\n",
    "# faded tick labels\n",
    "ax.tick_params(axis=\"both\", which=\"both\", labelsize=10)\n",
    "plt.setp(ax.get_xticklabels(), alpha=fade_alpha)\n",
    "plt.setp(ax.get_yticklabels(), alpha=fade_alpha)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "What do we learn from this exercise? Well, there's a big last mile issue with making narrative visualisations. Normally, when coding, you should do your best to avoid hard-coding numbers (for example `x=0.15` above). But in this case getting things just so requires a lot of manual adjustment and that's *after* you've decided what story you're going to tell, and how to show it.\n",
    "\n",
    "The example above demonstrates many of the pain points of the last mile, like decluttering, adjusting text, careful use of saturation and colour, and ensuring dates are displayed in an aesthetically pleasing way.\n",
    "\n",
    "One commonly used trick that we didn't see in the above example is labelling lines directly (rather than using a legend). So let's now see an example of this that was originally posted on the [Library of Statistical Translation](https://lost-stats.github.io/Presentation/Figures/line_graph_with_labels_at_the_beginning_or_end.html)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df = pd.read_csv(\n",
    "    \"https://raw.githubusercontent.com/LOST-STATS/LOST-STATS.github.io/master/Presentation/Figures/Data/Line_Graph_with_Labels_at_the_Beginning_or_End_of_Lines/Research_Nobel_Google_Trends.csv\",\n",
    "    parse_dates=[\"date\"],\n",
    ")\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "fade_alpha = 0.7\n",
    "\n",
    "# Create the column we wish to plot\n",
    "title = \"Log of Google Trends Index\"\n",
    "df[title] = np.log(df[\"hits\"])\n",
    "df = df.dropna(subset=[title])\n",
    "\n",
    "# Make a plot\n",
    "fig, ax = plt.subplots()\n",
    "\n",
    "# Add lines to it\n",
    "sns.lineplot(\n",
    "    ax=ax,\n",
    "    data=df,\n",
    "    x=\"date\",\n",
    "    y=title,\n",
    "    hue=\"name\",\n",
    "    palette=\"deep\",\n",
    "    legend=None,\n",
    "    hue_order=df[\"name\"].unique(),\n",
    "    alpha=fade_alpha,\n",
    ")\n",
    "# Add the text--for each line, find the end, annotate it with a label, and\n",
    "# adjust the chart axes so that everything fits on.\n",
    "for line, name in zip(ax.lines, df[\"name\"].unique()):\n",
    "    y = line.get_ydata()[-1]  # NB: to use start value, set [-1] to [0] instead\n",
    "    x = line.get_xdata()[-1]\n",
    "    if not np.isfinite(y):\n",
    "        y = next(reversed(line.get_ydata()[~line.get_ydata().mask]), float(\"nan\"))\n",
    "    if not np.isfinite(y) or not np.isfinite(x):\n",
    "        continue\n",
    "    text = ax.annotate(\n",
    "        name,\n",
    "        xy=(x, y),\n",
    "        xytext=(2, -2),\n",
    "        color=line.get_color(),\n",
    "        xycoords=(ax.get_xaxis_transform(), ax.get_yaxis_transform()),\n",
    "        textcoords=\"offset points\",\n",
    "        fontweight=\"bold\",\n",
    "    )\n",
    "    text_width = (\n",
    "        text.get_window_extent(fig.canvas.get_renderer())\n",
    "        .transformed(ax.transData.inverted())\n",
    "        .width\n",
    "    )\n",
    "    if np.isfinite(text_width):\n",
    "        ax.set_xlim(ax.get_xlim()[0], text.xy[0] + text_width * 1.05)\n",
    "\n",
    "# Title that gives the narrative\n",
    "plt.suptitle(\n",
    "    \"Economics overtakes other STEM subjects in searches\",\n",
    "    size=12,\n",
    "    ha=\"left\",\n",
    "    x=0.12,\n",
    ")\n",
    "# Horizontal y-axis title, faded\n",
    "ax.set_title(\n",
    "    title,\n",
    "    loc=\"left\",\n",
    "    size=10,\n",
    "    alpha=fade_alpha,\n",
    ")\n",
    "ax.set_xlabel(\"\")\n",
    "ax.set_ylabel(\"\")\n",
    "# remove chart clutter\n",
    "for key, spine in ax.spines.items():\n",
    "    spine.set_visible(False)\n",
    "ax.tick_params(axis=\"y\", which=\"both\", length=0)\n",
    "ax.tick_params(axis=\"x\", which=\"both\", color=[1, 0, 0, fade_alpha])\n",
    "# Format the date axis to be prettier.\n",
    "ax.xaxis.set_major_formatter(mdates.DateFormatter(\"%d %b\"))\n",
    "ax.xaxis.set_minor_locator(mdates.DayLocator())\n",
    "ax.xaxis.set_major_locator(mdates.AutoDateLocator(interval_multiples=False))\n",
    "# for time series, tick marks on the right help give a sense of right-ward motion\n",
    "ax.yaxis.tick_right()\n",
    "ax.grid(which=\"major\", axis=\"y\", lw=0.2)\n",
    "ax.set_ylim(0, None)\n",
    "plt.show()"
   ]
  }
 ],
 "metadata": {
  "celltoolbar": "Tags",
  "interpreter": {
   "hash": "671f4d32165728098ed6607f79d86bfe6b725b450a30021a55936f1af379a247"
  },
  "kernelspec": {
   "display_name": "Python 3.8.12 64-bit ('codeforecon': conda)",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}
